Add CI performance benchmark suite#223
Conversation
Performance results need a stable workload so source-data changes do not look like runtime regressions. Add a small committed project split across core, filesystem, and network modules for repeatable Git, ripgrep, zstd, and Make measurements. Keep the corpus separate because its contents define the benchmark and should change only when baselines are deliberately regenerated.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
c606264 to
899d081
Compare
899d081 to
3a0517a
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
3a0517a to
653c5fd
Compare
|
Once the benchmark is complete, upload the results to GitHub Pages, as rv32emu-bench does. |
653c5fd to
fb40607
Compare
fb40607 to
c5495eb
Compare
henrybear327
left a comment
henrybear327
left a comment
There was a problem hiding this comment.
In general, LGTM. Just some nits, as the text / comment is bloating a bit I feel.
Thanks.
| green `make check` covers BusyBox validation. Use `make test-busybox` to | ||
| iterate on a single applet failure without rerunning the unit suite. | ||
|
|
||
| ## Performance Benchmarks |
There was a problem hiding this comment.
nit: make the text concise
I think the current testing.md has grown to the size that's already too big and too complicated to read
I create elfuse-bench and @Max042004 can maintain it directly. |
Functional tests do not reveal changes in syscall, process-startup, or application-level performance. Add a two-tier suite so pull requests produce repeatable measurements and expose regressions before merge. Run pinned lmbench microbenchmarks and representative developer workloads with warmups and median reporting. Store raw samples in JSON, compare them with captured elfuse, QEMU Linux, and OrbStack baselines, and treat missing baseline metrics as failures. Run the suite exclusively on the self-hosted Apple Silicon machine to avoid interference from sibling jobs. Keep comparison report-only until runner variance is characterized, while retaining artifacts for review and baseline refreshes. Closes sysprog21#195
c5495eb to
c2c8cbe
Compare
|
Instead of using hard-to-understand filenames such as |
jserv
left a comment
jserv
left a comment
There was a problem hiding this comment.
Review of the CI benchmark suite. Findings are inline; the substantive risk is the host-lock lifecycle. The generated tests/bench-corpus/ modules were not line-reviewed.
Checked and found correct: the reader/writer flock logic never over-grants (shared is refused while an exclusive lock is held or pending, and the double writer_pending() check closes the publish race); bench-compare regression math and its baseline<=0 / division guards; threshold parsing; missing-metric-as-hard-failure under --fail-on-missing; bench-promote's atomic tempfile+fsync+os.replace and partial-capture provenance; and the workflow's EXIT-trap releases plus the always() net.
| if listener is not None: | ||
| listener.close() | ||
| try: | ||
| os.unlink(control_path) |
There was a problem hiding this comment.
Control-socket cleanup race that can orphan a live lock holder. The detached holder unconditionally os.unlink(control_path) here even on the clean, socket-driven release path (flock already dropped, released=True). The bench step in main.yml does release (holder A) immediately followed by acquire exclusive (holder B) reusing the same fixed CONTROL_FILE path. If holder A is descheduled -- exactly the busy-host scenario this lock targets -- so its unlink lands after holder B has re-bind()ed the recreated socket, A deletes B's socket. B's later EXIT-trap release then hits FileNotFoundError, reports nothing to release, and B keeps the exclusive flock until the 120-min self-expiry, wedging every sibling leg's shared acquire. Suggest guarding this unlink with if not released: so a successor holder that re-bound the same path is never clobbered.
| except (FileNotFoundError, ConnectionError, socket.timeout): | ||
| pass | ||
|
|
||
| for path in (pid_path, control_path): |
There was a problem hiding this comment.
Same class as the holder-side race: the release client unlinks the control socket unconditionally, including when it did not confirm a release. A socket.timeout or ConnectionError on connect/recv is swallowed above, then this loop removes a still-live holder's socket path -- if the holder was briefly stuck before accept(), it never gets the release, never drops its flock, and is now permanently unreachable for any future release (held until self-expiry). Suggest unlinking control_path only when released is true; leave a still-live holder's socket in place so a retry can reach it.
| prep="$prep && cat /bench-work/corpus/*/*.c > /bench-work/zin" | ||
| prep="$prep && i=0 && while [ \$i -lt $ZSTD_DOUBLINGS ]; do cat /bench-work/zin /bench-work/zin > /bench-work/zin.2 && mv /bench-work/zin.2 /bench-work/zin || exit 1; i=\$((i + 1)); done" | ||
| qemu_exec timeout "$TEST_TIMEOUT" sh -c "$prep" \ | ||
| > /dev/null 2>&1 \ |
There was a problem hiding this comment.
The qemu Tier-2 corpus prep (512-shard cp -R plus the zstd doublings) runs under timeout "$TEST_TIMEOUT" -- the per-sample bound -- and bench_dies the whole run on a slow-but-valid guest. The native (prepare_corpus) and orbstack prep paths do the same heavy copy work untimed, so the transports bound prep inconsistently and a legitimately slow guest FS becomes a hard failure rather than a measurement. Suggest bounding prep with its own (larger) timeout applied consistently across transports, separate from the per-sample TEST_TIMEOUT.
| # Tier-2 rows are bench-timeit's workload-only microseconds; report ms. | ||
| apps = data.get("applications", {}) | ||
| for bench, rows in sorted(apps.items()): | ||
| vals = rows[bench] |
There was a problem hiding this comment.
vals = rows[bench] assumes the metric key equals the bench key. Every current Tier-2 producer emits applications <metric> <metric> <value> so it holds today, but it is an unguarded invariant: any future row where bench != metric raises KeyError and crashes the JSON emitter with an opaque traceback instead of a diagnosable error. Suggest iterating rows.items() (emit each metric under bench) or asserting the invariant with a clear message.
| return 1; | ||
| } | ||
|
|
||
| printf("%llu\n", (unsigned long long) (elapsed / 1000)); |
There was a problem hiding this comment.
elapsed / 1000 floors nanoseconds to microseconds, so any workload finishing under 1 microsecond prints 0. Downstream summarize() accepts that as a valid fast sample and bench-compare scores it as a large improvement rather than a broken/instant measurement. Harmless for the current ms-scale corpus, but it removes the only guard against a workload that does no work reading as improved. Suggest emitting higher resolution (keep nanoseconds) or rejecting elapsed < 1000 as an invalid sample.
3 participants
Summary
Add a reproducible two-tier performance benchmark suite for detecting
regressions in pull requests.
filesystem latency.
against a deterministic committed corpus.
and OrbStack.
CI integration
Add a benchmark leg to the self-hosted Apple Silicon runtime matrix.
The leg runs exclusively on the host to avoid interference from sibling
jobs, compares results in report-only mode, and uploads the JSON artifact.
Missing baseline metrics are treated as failures, preventing broken or
timed-out benchmarks from silently passing. Benchmark preparation and
cleanup commands are also bounded by timeouts.
The checked-in baseline has not yet been captured on the designated
self-hosted CI machine. It will be refreshed in a follow-up after the
benchmark leg is deployed and runner variance has been characterized.
Comparison therefore remains report-only for now.
Validation
Closes #195
Summary by cubic
Adds a two-tier performance benchmark suite with a deterministic workload to catch syscall, process-startup, and app-level regressions. Runs on the self-hosted Apple Silicon runner under an exclusive host lock and posts a sanitized PR comment with benchmark deltas. (Issue #195)
New Features
tests/bench-suite.sh: Tier 1lmbench(lat_syscall/lat_proc/lat_fs) and Tier 2 app workloads (git status,rg,make,zstd) with warmups; guest-side timing forqemu-aarch64viatests/bench-timeit.c; writes medians+samples tobuild/bench-results*.json.tests/bench-corpusfor stable Tier 2 inputs; excluded from formatting to preserve baselines. Fixtures fetch app tools and cross-compile pinnedlmbench.tests/bench-baseline.jsonforelfuse-aarch64,qemu-aarch64, andorbstack.scripts/bench-compare.pyflags regressions (tunable viaBENCH_REGRESSION_THRESHOLD) and treats missing metrics as NEW by default;scripts/bench-promote.pyrefreshes baselines.make bench/make bench-ci;BENCH_ENVselectselfuse-aarch64(default) |qemu-aarch64|orbstack|native. Benchmark leg isolates measurements withscripts/ci-host-lock.sh..github/workflows/bench-report.ymlusesscripts/bench-report.pyto publish a sanitized advisory PR comment from the uploaded report artifact. Docs indocs/testing.md.Migration
make bench(orBENCH_ENV=qemu-aarch64 make bench-ci).scripts/bench-compare.py --report-only.make bench-ciper env, thenscripts/bench-promote.py; commit the updatedtests/bench-baseline.json.Written for commit c2c8cbe. Summary will update on new commits.